Automated generation of high-integrity test suites from graphical specifications

There is a strong need to ensure that the software controlling safety–critical systems is of the highest possible integrity. Present approaches to achieving this integrity through Verification and Validation (V&V) are expensive and predominantly reliant on manual processes. Techniques that have been proposed to improve the V&V of safety–critical systems – formal methods, graphical specifications and automated testing – do not fully address the needs of the industry. Formal methods have still to gain widespread use, graphical specifications are generally not well supported by rigorous V&V techniques and automation has not yet successfully addressed the problem of specification–based test design in a rigorous yet flexible manner. This thesis describes the integration of graphical specification notations, formal methods and automated testing to achieve efficient and effective V&V. A framework is presented within which formal and automated techniques can be applied to the validation of graphical specifications and the automated generation of effective tests from these specifications. A formalisation of two notations of proven practical use (Statecharts and PFS tables) is presented. An intermediate formal representation of the specifications then forms the basis of a set of generic automated V&V techniques. Common structures in the formal specification are exploited to develop automated specification validation techniques based on automated theorem proving and model checking. Automated test case design is achieved by formally specifying testing heuristics, based on either partition testing strategies or particular hypothesised faults. These heuristics are then automatically applied to the formal representation to produce the test cases. In some situations, test cases must be sequenced in order to indirectly infer the correctness of parts of the system not directly measurable at the testing interface. The formalisation is extended to provide an abstraction that allows existing finite state machine-based techniques for selecting test sequences to be exploited. Methods for optimising the effectiveness and efficiency of the testing heuristics are described as are industrially applicable case studies that validate the techniques. The thesis concludes that the formalisation of specifications and testing heuristics can lead to a practical level of automation that not only has the potential to reduce the costs involved in V&V but can also increase its effectiveness, allowing a greater number of defects to be detected earlier in the software development.